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Abstract 

Safety  research  in  high-reliability  industries,  such  as  aviation,  has  clearly  shown 
that  the  causes  of  accidents  are  primarily  related  to  deficiencies  in  nontechnical 
(cognitive  and  social)  skills,  rather  than  a  lack  of  technical  expertise. 
Investigations  into  causes  of  error  in  anesthesia  have  revealed  a  similar  pattern. 
The  aviation  industry  uses  behavioral  marker  systems  to  structure  the  training  and 
assessment  of  pilot  crew  resource  management  (CRM)  (nontechnical)  skills. 
These  are  empirically  derived  taxonomies  of  the  principal  required  nontechnical 
skills,  and  an  observation-based  rating  system  for  assessing  their  component 
behaviors.  In  medicine,  the  enthusiasm  for  adopting  CRM-type  training  needs  to 
be  underpinned  by  the  development  of  a  properly  tested  nontechnical  skills 
framework  for  a  given  specialization.  A  research  project  involving  industrial 
psychologists  and  consultant  anesthetists,  and  funded  by  the  National  Health 
Service  Education  for  Scotland,  is  developing  and  evaluating  a  behavioral  marker 
system  for  anesthetists.  It  has  produced  and  experimentally  evaluated  a  taxonomy 
of  anesthetists’  nontechnical  skills  (ANTS),  which  can  be  used  to  guide 
professional  syllabi,  offer  a  method  for  evaluating  performance,  and  improve  the 
quality  of  feedback  to  trainees.  This  paper  will  provide  an  overview  of  the  ANTS 
System  and  the  results  of  usability  trials  conducted  in  the  operating  theatre. 


Introduction 

Safety  research  in  high-reliability  industries,  such  as  aviation,  has  shown  that 
accidents  frequently  involve  deficiencies  in  nontechnical  skills  (cognitive  skills 
such  as  decisionmaking  or  social  skills  such  as  teamworking).1  There  is  increasing 
awareness  of  a  similar  pattern  in  health  care.  Critical  incident  studies  in  anesthesia 
have  found  that  around  80  percent  of  reported  incidents  involve  human  factors 

2,  3 

causes. 

The  aviation  industry  responded  to  this  problem  with  the  development  of 
Crew  Resource  Management  (CRM)  training  programs  to  provide  training  in 
nontechnical  skills  and  promote  safety  behavior  on  the  flight  deck.1' 4' 5  Thereafter, 
behavioral  marker  systems  were  developed  to  permit  structured  observation  and 
assessment  of  these  skills.4  A  behavioral  marker  system  is  an  empirically  derived 
taxonomy  of  the  principal  nontechnical  skills  required  in  a  particular  domain,  and 
an  observation-based  rating  system  for  assessing  component  behaviors.  Explicit 
training  in  nontechnical  skills  for  anesthesia  has  previously  been  limited; 
however,  with  the  advent  of  high-fidelity  patient  simulators,  several  CRM -type 
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programs  are  being  developed.  ’  To  be  effective,  such  training  needs  to  be 
reinforced  by  explicit  and  constructive  feedback,  both  at  the  simulator  and  during 
regular  training  in  the  hospital.  The  Anesthetists’  Nontechnical  Skills  (ANTS) 
Project  in  Scotland,  involving  industrial  psychologists  and  consultant  attending 
anesthesiologists,  sought  to  develop  a  behavioral  marker  system  for  use  in 
anesthesia  training,  both  at  the  simulator  and  on  the  job.8 

A  bottom-up  approach  was  adopted  for  development  of  the  ANTS  System.9 
Three  sources  of  data  were  used  to  identify  a  full  set  of  anesthetists’  nontechnical 
skills.  First,  the  current  state  of  knowledge  regarding  nontechnical  skills  and  the 
use  of  behavioral  marker  systems  in  anesthesia  was  established,  with  a  review  of 
human  factors  research.10  Cognitive  task  analysis  interviews  were  conducted  with 
29  consulting  anesthetists  by  a  psychologist  with  a  basic  knowledge  of 
anesthesia.9' 11  The  anesthetists  were  asked  to  recall  and  describe  their 
management  of  a  challenging  case  and  to  list  the  skills  they  considered  important 
for  good  practice  in  anesthesia.  The  resulting  data  were  analyzed  utilizing  a 
grounded  theory  approach  to  identify  the  nontechnical  issues.12  Information  from 
a  survey  of  anesthetists’  attitudes  toward  safety  and  teamwork  was  also  taken  into 
account.13 

From  these  data,  a  prototype  behavioral  marker  system  was  developed  by  the 
design  team  according  to  a  number  of  design  criteria:  the  skills  should  be 
observable;  the  system  should  have  a  hierarchical  structure  and  should  be  simple 
and  usable  with  minimal  training;  and  it  should  be  complementary  to  the 
competency-based  approach  being  adopted  in  medical  education  in  the  United 
Kingdom.  The  resulting  prototype  was  then  refined  and  condensed  iteratively  by 
making  observations  in  the  operating  room,  recoding  a  sample  of  the  interview 
transcripts,  and  reviewing  200  anesthesia  critical-incident  reports  to  cross-check 
that  all  the  nontechnical  issues  associated  could  be  explained.  After  the 
production  of  a  final  prototype,  examples  of  good  and  poor  behaviors  were 
selected  for  each  element. 

Lessons  from  the  aviation  industry  highlight  the  need  for  such  systems  to  be 
fully  evaluated  before  use  and  then  introduced  through  a  well-supported  process 
for  successful  integration  into  existing  training  programs.4  So  the  next  stage  of  the 
project  was  an  experimental  evaluation  of  the  ANTS  System.  Video  of  simulated 
anesthesia  cases  showed  that  even  with  minimal  training  of  the  raters,  there  were 
sufficient  levels  of  validity,  reliability,  and  usability  to  allow  further  testing  for 

o 

usability  in  the  real  training  environment.  User  trials  are  a  standard  human 
factors  research  method  and  are  extensively  used  to  investigate  usability  issues.14 
In  order  to  conduct  a  trial  to  investigate  the  usability  issues  associated  with  real- 
life  training  environments,  it  was  necessary  to  identify  the  implementation 
requirements,  such  as  user  training,  and  develop  user  materials  and  guidance.  This 
paper  reports  the  finding  of  this  stage  of  the  project. 
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The  ANTS  System 

The  ANTS  System  is  a  behavioral  marker  system  with  a  hierarchical  structure 
starting  with  four  main  categories:  task  management,  teamworking,  situation 
awareness,  and  decisionmaking.  These  categories  are  subdivided  into  a  total  of  15 
skill  elements  that  form  the  main  framework  of  the  system  (Table  1). 


Table  1.  The  ANTS  System:  categories  and  elements 


Category 

Element 

Task  management 

Planning  and  preparing 

Prioritizing 

Providing  and  maintaining  standards 

Identifying  and  utilizing  resources 

Teamworking 

Coordinating  activities  with  team  members 
Exchanging  information 

Using  authority  and  assertiveness 

Assessing  capabilities 

Supporting  others 

Situation  awareness 

Gathering  information 

Recognizing  and  understanding 

Anticipating 

Decisionmaking 

Identifying  options 

Balancing  risks  and  selecting  options 

Reevaluating 

Each  element  has  a  definition  and  sample  behavioral  markers  (Table  2).  These 
provide  some  examples  of  good  and  poor  behaviors  that  could  be  associated  with 
each  skill  element. 


Table  2.  Examples  of  behavioral  markers  for  teamworking 


Cateaorv:  Teamworkinq 

Element  -  Coordinating  activities  with  team  members 

Behavioral  markers  for  good  practice 

Behavioral  markers  for  poor  practice 

•  Confirms  roles  and  responsibilities  of 

•  Does  not  coordinate  with  surgeons 

team  members 

and  other  groups 

•  Discusses  care  with  surgeons  or 

•  Relies  too  much  on  familiarity  of 

colleagues 

team  for  getting  things  done 

•  Considers  requirements  of  others 

•  Intervenes  without 

before  acting 

informing/involving  others 

•  Cooperates  with  others  to  achieve 

•  Does  not  involve  team  in  tasks 

goals 

The  system  was  not  designed  to  be  fully  exhaustive  and  is  limited  to  the 
principal  skills  that  can  be  identified  through  observable  behavior.  Hence  stress 
management  and  communication  are  not  included  in  the  system;  stress 
management  is  difficult  to  observe  and  communication  is  frequently  the  means  by 
which  the  other  skills  are  observed. 
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Assessment  can  be  made  at  both  the  element  and  category  levels.  A  rating 
scale  was  developed  with  the  system  and  was  used  during  the  experimental 
evaluation.  This  has  four  points  for  describing  level  of  performance  with  an 
additional  option  for  nonobserved  skills  (Table  3).  The  nonobserved  skills  option 
is  available  for  use  in  circumstances  when  it  is  not  appropriate  for  an  element  to 
be  demonstrated. 


Table  3.  ANTS  System  rating  options 


Rating  Label 

Description 

4 — Good 

Performance  was  of  a  consistently  high  standard,  enhancing 
patient  safety;  it  could  be  used  as  a  positive  example  for  others. 

3 — Acceptable 

Performance  was  of  a  satisfactory  standard  but  could  be  improved. 

2 — Marginal 

Performance  indicated  cause  for  concern;  considerable  improvement 
is  needed. 

1 — Poor 

Performance  endangered  or  potentially  endangered  patient  safety; 
serious  remediation  is  required. 

Not  observed 

Skill  could  not  be  observed  in  this  scenario. 

Methods 

Questionnaire 

Based  on  results  from  the  previous  evaluation  study,  a  questionnaire  was 
designed  to  address  more  specific  user  issues.  This  questionnaire  was  sent  to  the 
50  consultant  anesthetists  who  had  participated  in  that  part  of  the  project.  The 
questionnaire  was  returned  anonymously. 

Operating  room  user  trials 

We  planned  to  conduct  these  trials  during  routine  training  in  both  the 
operating  room  and  simulator  training  courses  for  trainee  anesthetists.  However, 
the  limited  time  for  this  stage  of  the  project  and  changes  in  the  face  of  simulator 
training  in  Scotland,  where  there  is  increased  involvement  of  other  medical 
specialties,  restricted  the  investigation  to  the  operating  room  environment. 

An  invitation  to  participate  in  the  user  trial  was  sent,  along  with  the 
questionnaire,  to  the  50  consultant  anesthetists  who  had  taken  part  in  the 
evaluation  study.  Twenty  consultants  agreed.  During  the  trial,  participating 
consultants  recruited  trainee  anesthetists,  who  agreed  to  be  evaluated  using  the 
ANTS  System  during  scheduled  training  lists.  In  addition  to  obtaining  ethical 
approval  from  the  Multi-centre  Research  Ethics  Committee  and  the  appropriate 
local  research  ethics  committees,  the  clinical  directors  of  anesthesia  in  each 
hospital  and  the  directors  of  anesthesia  training  programs  were  informed  about  the 
study. 

The  consultants  attended  a  3-hour  training  session  to  revise  the  ANTS  System, 
giving  them  the  opportunity  for  further  practice  in  rating  and  discussing  these 
ratings  with  their  colleagues.  Information  about  the  purpose  and  procedure  for  the 
trials  was  also  distributed  and  discussed.  The  consultants  were  provided  with 
briefing  notes,  instructions  for  the  user  trials,  a  copy  of  a  small  handbook  with 
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details  of  the  ANTS  System,  and  copies  of  a  feedback  questionnaire  to  be 
completed  for  each  session  in  which  they  used  the  ANTS  System.15  Consultants 
were  required  to  inform  the  trainees  about  the  proposed  use  of  the  ANTS  System 
before  the  session.  The  trainees,  who  were  given  feedback  using  the  ANTS 
System,  were  also  given  an  optional  feedback  questionnaire  to  complete,  which 
they  returned  anonymously. 


Results 

Questionnaire 

Thirty-one  of  50  consultants  (62  percent)  returned  questionnaires.  All  of  the 
respondents  said  they  thought  the  ANTS  System  would  be  useful  to  support 
training  in  anesthesia,  although  only  2 1  consultants  reported  having  referred  to  the 
system  since  the  evaluation  study  (9  months  earlier).  They  reported  using  the 
ANTS  System  informally  to  support  teaching,  as  a  guide  to  self-reflection,  and 
discussion  of  the  system  with  colleagues  and  trainees. 

Responses  indicated  that  the  consultants  thought  the  system  could  be 
employed  usefully  at  all  stages  of  training.  The  consultants  were  asked  to  rate  the 
usefulness  of  the  system  (Scale:  1  =  No  use;  2  =  Limited  use;  3  =  Useful;  4  = 

Very  useful;  5  =  Extremely  useful).  Respondents  thought  the  system  would  be 
most  useful  for  structuring  observations  and  assessments  to  give  feedback  (90 
percent,  n  =  34),  and  could  also  be  useful  for  structuring  formal  periodic 
assessments  (65.5  percent,  n  =  20),  for  self-reflection  (65.5  percent,  n  =  20),  and 
structuring  teaching  (52  percent,  n  =  16)  (Table  4).  It  was  thought  that  the  highest 
utility  would  be  in  simulation  centers. 


Table  4.  Training  applications  for  the  ANTS  System  suggested  by  consultants 


Application 

Frequency 

Comment 

For  structuring  teaching 

52% 

As  a  framework  or  for  planning;  limited  use. 

For  structuring  observations 
and  assessments  for  feedback 

90% 

As  a  language/framework  for  discussion; 
makes  observations  more  focused. 

For  structuring  formal  periodic 
assessments 

65.5% 

After  the  system  has  undergone  further 
evaluation;  as  summary  only;  as  framework. 

For  structuring  self-reflection 

65.5% 

Way  of  thinking  about  task;  as  input;  not  sure. 

Other  (please  specify) 

21% 

For  assessment  and  giving  feedback;  only  if 
video  recordings  are  available;  to 
debrief/discuss  critical  incidents/near  misses; 
encourage  new  ways  of  thinking  about  task. 

*  Total  number  of  respondents  =  31 


All  of  the  respondents  had  previously  attended  a  4-hour  training  session  as 
part  of  the  evaluation  study  but  felt  that  further  familiarity  with  the  system  would 
be  necessary  if  they  were  to  use  it  comfortably  and  reliably  in  real  training 
settings.  Suggestions  made  for  additional  guidance  material  and  training 
included — 

•  A  simple  handbook  for  use  in  the  theater. 
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•  Refresher  training  with  further  practice,  similar  to  the  evaluation  study 
but  also  allowing  more  discussion  of  observations  and  ratings  with 
colleagues,  to  pennit  calibration  of  standards. 

•  Frequent  use  of  the  system  to  stay  current. 

Asked  if  they  could  anticipate  any  problems  with  using  the  ANTS  System  in  a 
real  training  setting,  only  one  respondent  answered  in  the  affirmative.  Others 
offered  comments  noting  time  constraints  to  practice  and  gain  familiarity  with  the 
ANTS  System,  and  difficulty  to  use  if  the  consultant  is  heavily  involved  in  the 
case  or  if  there  is  an  incident.  Respondents  thought  it  would  not  be  possible  to  use 
the  system  when  personally  directing  patient  care,  in  an  emergency,  or  when 
caring  for  a  very  sick  patient.  It  would  also  be  difficult  with  very  inexperienced 
trainees,  during  lists  requiring  a  rapid  turnover  of  cases,  and  in  other  specified 
circumstances  that  were  considered  poor  for  training  purposes. 

Consultants  (n  =  21)  commented  on  how  they  thought  the  ANTS  System 
could  be  best  implemented  to  support  nontechnical  skills  training  in  anesthesia. 
They  felt  that — 

•  The  ANTS  System  needs  to  be  introduced  carefully,  to  guide 
reflection  or  to  assist  consultants  on  a  personal  choice  basis. 

•  A  few  consultants  (for  each  anesthetic  department)  need  to  be  trained 
to  use  the  system  for  assessment. 

•  ANTS  should  be  used  mainly  in  the  simulator  environment,  but  could 
also  be  used  to  an  extent  for  routine  elective  cases. 

Operating  room  user  trials 

A  total  of  38  feedback  forms  were  received  from  17  consultant  users  at  5 
different  hospitals.  One  feedback  form  was  completed  on  each  occasion  the 
ANTS  System  was  used.  Different  participants  returned  different  numbers  of 
forms  (mean  return  =  2,  mode  =  1,  range  =  1-7).  A  total  of  24  feedback  forms 
were  received  from  the  anesthetists  in  training  for  the  38  occasions  the  ANTS 
System  was  used  (63  percent  trainee  response  rate). 

Overall,  the  system  was  reported  to  have  enhanced  training  by  both 
consultants  (79  percent  of  feedback  fonns,  n  =  30)  and  trainees  (83  percent  of 
feedback  forms,  n  =  20).  Consultants  felt  use  of  the  ANTS  System  was  beneficial 
because  it  provided  a  framework  for  a  discussion,  where  feedback  might 
previously  have  been  missed.  Both  trainees  and  consultants  were  able  to  reflect  on 
and  reinforce  what  makes  good  practice.  A  few  trainees  specifically  commented 
that  use  of  the  system  had  generally  raised  their  awareness  of  nontechnical  skills 
(n  =  3),  that  they  had  benefited  from  receiving  feedback  on  their  performance 
(n  =  6),  and  that  the  system  helped  their  self-reflection  (n  =  2). 

The  system  was  used  with  trainees  at  all  training  levels,  with  approximately 
equal  numbers  of  senior  house  officers  (trainees  in  first  or  second  year),  and 
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specialist  registrars  (trainees  in  third  to  sixth  year)  (45  percent  and  50  percent, 
respectively).  The  grade  of  trainee  was  not  reported  in  5  percent  of  cases. 

Consultants  and  trainees  were  asked  to  indicate  how  they  felt  the  ANTS 
System  had  been  used  in  each  session  and  rate  this  approach  on  a  five-point  scale 
from  no  use  to  extremely  useful  (Table  5).  The  three  uses  that  particularly  stood 
out  for  the  consultants  were  as  a  debriefing/feedback  framework,  for  postsession 
rating  of  skills,  and  as  a  language  for  discussion.  Consultants  were  asked  to  detail 
whether  they  thought  the  ANTS  System  was  useful  for  rating  skills  concurrently 
or  after  a  session  in  order  to  gain  insight  as  to  what  stage  it  was  most  usable  for 
assessment  of  performance.  Trainees  were  only  asked  if  they  considered  the 
system  useful  for  rating  skills  in  general.  As  with  the  consultants,  use  of  the 
system  for  debriefing  and  to  provide  a  common  language  for  discussion  scored 
highly  for  the  trainees.  They  also  felt  that  the  consultants  were  using  the  system  to 
describe  their  observations,  which  they  rated  between  useful  and  very  useful,  but 
they  rated  the  system  most  useful  as  a  tool  for  self-reflection. 


Table  5.  Uses  of  the  ANTS  System 


Possible  uses  of 
ANTS  System 

Consultants 
(38  feedback  forms) 

Trainees 

(24  feedback  forms) 

%  used 

Mean  rating*  +  SD 
Range 

%  used 

Mean  rating  +  SD 
Range 

Language  for 
discussion 

87% 

3.6  +  0.79 

2-5 

83% 

3.7  +  0.72 

3-5 

Briefing 

framework 

53% 

3.2  +  0.72 

2-4 

62% 

3.3  +  0.9 

2-5 

Teaching 

framework 

68% 

3.5  +  0.76 

2-5 

71% 

3.9  +  0.7 

3-5 

To  describe 
observations 

71% 

3.5  +  0.97 

2-5 

83% 

3.7  +  0.8 

2-5 

To  rate  skills 
a)  concurrent 

a)  postsession 

58% 

92% 

3.0  +  0.81 

2-5 

3.6  +  0.69 

2-5 

75% 

3.6  +  0.86 

2-5 

Debriefing 

framework 

89% 

4.1  +  0.81 

2-5 

92% 

4.0  +  0.65 

3-5 

Self-reflection 

tool 

79% 

3.6  +  0.93 

2-5 

87% 

4.1  +  0.65 

3-5 

*  Rating  scale:  1=No  use;  2=Limited  use;  3=Useful;  4=Very  useful;  5=Extremely  useful 
SD  =  standard  deviation 


The  ANTS  System  was  used  in  its  entirety  89  percent  of  the  time.  The 
majority  of  consultants  (89  percent,  n  =  33)  also  reported  that  they  used  it  at  both 
category  and  element  levels.  The  elements  seemed  to  be  most  useful  for  providing 
specific  detailed  feedback  about  strengths  and  weaknesses,  whereas  the  categories 
were  useful  for  giving  an  overview  and  as  a  tool  for  more  general  discussions. 

It  was  common  for  the  system  to  be  used  with  the  trainee  leading  a  case  and 
the  consultant  taking  a  secondary  role  to  make  observations,  but  available  to  step 
in  and  take  over  if  the  clinical  situation  required.  One  consultant  reported  it  was 
difficult  to  use  the  ANTS  System  when  the  consultant  was  required  to  lead  the 
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case.  A  different  approach  used  with  a  trainee  who  was  insufficiently  experienced 
to  take  the  lead  in  the  case  was  to  use  the  ANTS  System  as  a  general  framework, 
with  discussions  of  nontechnical  skills  taking  place  during  the  case  as  issues 
arose. 

The  consultants  reported  using  their  ANTS  System  observations  and  ratings  to 
give  feedback  to  the  trainee — reviewing  strengths  and  weaknesses,  reinforcing 
good  behavior,  and  identifying  specific  ways  for  the  trainee  to  develop.  The 
system  was  also  reported  as  a  useful  way  to  encourage  senior  trainees  to  think 
about  the  nontechnical  skills  of  others. 

Thirteen  consultants  reported  making  notes  during  or  immediately  after  the 
case.  This  was  seen  as  important,  particularly  if  there  was  a  delay  between  the 
observation  session  and  debriefing.  Ratings  were  most  commonly  made  after  the 
case. 

Many  of  the  trainees  reported  being  given  the  ANTS  booklet  in  advance  or 
receiving  a  presession  briefing  on  the  training  aims  of  using  the  ANTS  System. 
One  trainee  who  had  not  been  fully  briefed  reported  confusion  about  the 
observation  and  rating  process  until  the  debriefing  session.  Comments  from  the 
trainees  strongly  suggest  that  they  found  it  beneficial  to  receive  feedback  on  their 
nontechnical  skills. 

With  regard  to  difficulties  in  using  the  system,  one  of  the  consultants 
identified  barriers  to  using  the  system  on  two  occasions,  but  not  on  two  other 
occasions.  The  problems  identified  by  others  were  the  need  to  stop  using  the 
system  due  to  changes  in  the  clinical  situation  (n  =  5),  having  to  assist  directly 
with  patient  care  (n  =  3),  and  lack  of  time  available  for  a  debriefing  (n  =  2). 
Consultants  also  found  it  difficult  not  to  be  involved  in  the  case  themselves  in 
order  to  rate  the  trainee  (n  =  2). 

Trainees  were  also  asked  if  they  could  imagine  training  situations  when  it 
would  not  be  possible  or  appropriate  to  use  the  ANTS  System.  They  reported  that 
it  would  not  be  appropriate  when  a  trainee  was  inexperienced  in  that  situation 
(n  =  2);  during  rapidly  evolving  emergencies  where  it  might  cause  additional 
stress  (n  =  1);  and  when  there  were  several  members  of  the  anesthesia  team, 
making  it  difficult  to  identify  the  behaviors  of  individuals  (n  =  2).  Two  trainees 
gave  criteria  for  when  they  thought  the  system  should  be  used  to  avoid  problems. 
For  instance,  they  thought  the  trainee  should  be  acting  as  lead  anesthetist  to  allow 
the  consultant  to  make  observations  independently,  and  that  it  could  be  used  in 
totally  new  situations  to  maximize  the  chance  of  seeing  all  the  elements 
demonstrated. 


Discussion 

The  user  trials  indicate  the  ANTS  System  can  be  used  in  a  real  training 
environment  to  give  fonnative  feedback,  providing  that  consultants  have  been 
trained  to  use  the  system,  suitable  cases  can  be  selected,  and  time  is  available  for 
postsession  debriefing.  This  is  acceptable  to  both  consultants  and  trainees. 
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The  most  commonly  employed  method  of  using  the  ANTS  System  involved 
the  consultant  anesthetist  observing  the  trainee  during  a  case  and  making  ratings 
once  the  case  was  over.  This  was  most  successful  when  the  trainee  could  take  the 
lead  role  either  throughout  or  for  a  particular  segment  of  a  case,  permitting  the 
consultant  to  make  independent  observations.  For  this  method,  it  is  crucial  that  the 
chosen  clinical  situation  is  appropriate  to  the  clinical  knowledge  and  experience 
of  the  trainee.  Ratings  are  ideally  made  after  the  case  is  over,  when  the  whole 
performance  can  be  analyzed.  Making  observation  notes  was  found  to  be 
particularly  helpful  when  there  was  a  delay  between  the  case  and  the  feedback 
session.  The  feedback  session  was  considered  critical  by  both  consultants  and 
trainees. 

In  the  majority  of  circumstances,  the  whole  ANTS  System  was  used  rather 
than  focusing  on  specific  areas.  However,  the  feedback  naturally  concentrated  on 
those  areas  considered  to  be  most  in  need  of  attention.  The  users  found  the 
element  level  was  preferred  over  the  categories  level  when  giving  detailed 
feedback,  and  also  that  the  system  was  easier  to  use  at  this  level.  This  was 
contrary  to  the  finding  of  the  evaluation  study,  where  the  trial  participants 

o 

perceived  the  category  ratings  easier  to  produce.  This  was  probably  because  the 
elements  were  rated  first  in  the  evaluation  study  and  this  assisted  in  making  the 
category  ratings,  or  it  may  be  that  behaviors  are  easier  to  observe  in  the  real 
situation  rather  than  on  a  video  as  the  rater  has  a  more  complete  understanding  of 
the  situation  being  rated.  As  the  ANTS  System  is  designed  as  an  observation- 
based  tool,  it  is  probably  best  to  make  ratings  at  the  element  level  first,  as  these 
can  be  linked  to  the  observed  behaviors.  This  will  also  facilitate  detailed 
feedback.  Category  ratings  may  then  be  derived,  if  necessary. 

The  sample  size  was  small,  and  this  probably  resulted  from  the  tight  time 
frame  available  for  training,  use  of  the  system,  and  then  returning  of 
questionnaires.  Those  who  found  difficulty  or  were  unable  to  use  the  ANTS 
System  reported  lack  of  opportunity  as  the  main  challenge,  in  that  they  were 
either  not  assigned  trainees  or  the  cases  were  not  suitable.  Another  reason  was 
lack  of  time  to  use  the  system. 

The  system  seemed  to  work  for  all  training  grades,  although  it  was  used 
slightly  more  often  with  more  experienced  trainees.  Perhaps  this  was  because 
these  trainees  had  the  appropriate  experience  for  a  wider  range  of  cases,  and  were 
able  to  lead  either  the  whole  case  or  a  specific  segment.  It  could  be  used  more 
often  as  a  discussion  tool  when  a  trainee  does  not  have  the  relevant  experience  to 
take  the  lead.  This  could  be  done  either  as  a  case  proceeded  or  during  a 
postsession  discussion.  The  trainee  responses  emphasized  their  need  to  be  familiar 
with  the  ANTS  System  before  it  is  used  to  provide  feedback.  Introduction  of 
ANTS  throughout  the  anesthesia  curriculum  might  permit  this  as  a  framework  for 
discussing  nontechnical  skills  in  the  early  stages  of  anesthesia  education,  which 
would  progress  to  more  observation  and  assessment  for  feedback  as  the  trainee 
gained  experience. 

There  were  no  problems  identified  with  the  supporting  material  provided  for 
the  participants  (i.e.,  observation  and  rating  forms  and  the  ANTS  handbook).  As 
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anticipated,  in  the  followup  questionnaires,  some  difficulties  were  encountered 
while  using  the  system,  but  none  of  these  were  considered  to  be  serious.  It  proved 
difficult  for  consultants  to  make  observations  and  ratings  of  a  trainee  while 
participating  directly  in  patient  management.  Trainees  also  felt  that  this  was  a 
situation  when  it  would  not  be  appropriate  to  use  the  system  as  an  assessment 
tool.  However,  this  problem  is  not  unique  to  the  ANTS  System,  but  is  the  case 
with  all  kinds  of  teaching  activity  and  would  not  prevent  discussion  of 
nontechnical  skills  issues  as  they  arise  in  a  case.  Alternatively  a  trainee  may  be 
able  to  take  the  lead  in  some  specific  segments  of  the  case,  which  could  be  an  area 
for  assessment. 

The  main  barrier  to  use  of  the  system  seemed  to  be  difficulties  in  securing 
enough  time  with  trainees  to  permit  a  proper  feedback  system.  Time  for  feedback 
must  be  scheduled  to  allow  effective  utilization  of  the  system.  Time  constraints 
operate  throughout  the  training  process  and  will  become  more  onerous  with  more 
rigid  limits  on  both  trainee  and  consultant  timetables.16 

A  major  driver  to  initiate  the  ANTS  Project  was  to  develop  a  validated 
behavioral  marker  system  to  support  debriefing  during  anesthesia  simulation 
training,  and  to  allow  the  assessment  of  perfonnance  to  measure  the  impact  of 
nontechnical  skills  training  in  the  future.  Results  from  the  followup  survey  of 
consultants  who  participated  in  the  evaluation  study  indicated  that  this  group  felt 
the  system  would  be  extremely  useful  during  simulation  training.  The  planned 
simulator  limb  of  the  user  trials  was  not  possible  due  to  time  constraints  and 
changes  in  the  pattern  of  specialty  training  in  the  simulator  in  Scotland.  Despite 
this,  the  system  has  been  used  in  this  setting,15  and  several  simulation  centers  in 
other  countries  are  currently  in  communication  with  the  project  team  regarding 
use  and  evaluation  of  the  system  in  their  own  training  programs. 

It  is  important  that  consultant  trainers  are  familiar  enough  with  the  ANTS 
System  to  use  it  easily  and  confidently.  In  addition,  they  require  skills  in  giving 
feedback  to  trainees  and  giving  concrete  advice  to  those  in  whom  weaknesses  are 
identified.  A  critical  factor  in  the  implementation  of  a  behavioral  marker  system 
such  as  the  ANTS  System  is  training  of  the  proposed  users  of  the  system.17  The 
consultants  taking  part  in  the  user  trials  had  a  total  of  7  hours  of  training  on  the 
ANTS  System.  The  initial  4  hours  of  training  took  place  during  the  evaluation 
study,  but  this  occurred  up  to  a  year  before  the  user  trials  began.  In  addition, 
design  of  the  evaluation  study  had  not  allowed  for  any  discussion  between  the 
consultants  regarding  their  observations  and  ratings.  It  was  clear  from  responses 
to  the  questionnaires  that  consultants  felt  they  required  further  training  before 
using  the  system  and  that  time  for  discussion  of  ratings  be  incorporated.  Clearly, 
the  training  time  is  not  enough,  and  a  minimum  of  2  consecutive  days  of  specialist 
training —  which  would  include  information  on  sources  of  rater  biases  and 
debriefing17 — would  be  recommended  for  use  of  a  behavioral  marker  system  such 
as  the  ANTS  System.  Ideally  many  video  examples  would  be  taken  from  clinical 
practice;  however,  there  are  ethical  and  practical  difficulties  filming  actual  events. 
For  this  project,  semiscripted  scenarios  were  acted  and  filmed  in  the  simulator, 
and  this  proved  acceptable  to  the  study  participants. 
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The  success  of  this  work  has  now  led  to  a  similar  investigation  to  develop  a 

1 8 

behavioral  marker  system  for  the  identification  of  surgeons’  nontechnical  skills. 


Conclusion 

This  practical  evaluation  of  the  ANTS  System  has  provided  infonnation  on 
usability  of  the  tool  in  the  real-world  training  setting,  albeit  with  a  small  sample. 
With  proper  training  of  raters,  briefing  of  trainees,  and  careful  case  selection,  the 
ANTS  System  may  offer  a  valuable  method  for  facilitating  structured  observation 
and  feedback  of  anesthesia  trainees’  nontechnical  skills.  It  is  recommended  that 
the  next  stage  of  evaluation  research  should  take  the  form  of  a  larger  study  of  both 
in-theatre  and  simulator  training  sessions. 
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